Evaluation of stacked ensemble model performance to predict clinical outcomes: A COVID-19 study.

BACKGROUND: The application of machine learning (ML) to analyze clinical data with the goal to predict patient outcomes has garnered increasing attention. Ensemble learning has been used in conjunction with ML to improve predictive performance. Although stacked generalization (stacking), a type of heterogeneous ensemble of ML models, has emerged in clinical data analysis, it remains unclear how to define the best model combinations for strong predictive performance. This study develops a methodology to evaluate the performance of "base" learner models and their optimized combination using "meta" learner models in stacked ensembles to accurately assess performance in the context of clinical outcomes. METHODS: De-identified COVID-19 data was obtained from the University of Louisville Hospital, where a retrospective chart review was performed from March 2020 to November 2021. Three differently-sized subsets using features from the overall dataset were chosen to train and evaluate ensemble classification performance. The number of base learners chosen from several algorithm families coupled with a complementary meta learner was varied from a minimum of 2 to a maximum of 8. Predictive performance of these combinations was evaluated in terms of mortality and severe cardiac event outcomes using area-under-the-receiver-operating-characteristic (AUROC), F1, balanced accuracy, and kappa. RESULTS: The results highlight the potential to accurately predict clinical outcomes, such as severe cardiac events with COVID-19, from routinely acquired in-hospital patient data. Meta learners Generalized Linear Model (GLM), Multi-Layer Perceptron (MLP), and Partial Least Squares (PLS) had the highest AUROC for both outcomes, while K-Nearest Neighbors (KNN) had the lowest. Performance trended lower in the training set as the number of features increased, and exhibited less variance in both training and validation across all feature subsets as the number of base learners increased. CONCLUSION: This study offers a methodology to robustly evaluate ensemble ML performance when analyzing clinical data.

Epidemic Vulnerability Index for Effective Vaccine Distribution Against Pandemic

As COVID-19 vaccines have been distributed worldwide, the number of infection and death cases vary depending on the vaccination route. Therefore, computing optimal measures that will increase the vaccination effect are crucial. In this paper, we propose an Epidemic Vulnerability Index (EVI) that quantitatively evaluates the risk of COVID-19 based on clinical and social statistical feature analysis of the subject. Utilizing EVI, we investigate the optimal vaccine distribution route with a heuristic approach in order to maximize the vaccine distribution effect. Our main criterias of determining vaccination effect were set with mortality and infection rate, thus EVI was designed to effectively minimize those critical factors. We conduct vaccine distribution simulations with nine different scenarios among multiple Agent-Based Models that were constructed with real-world COVID-19 patients’ statistical data. Our result shows that vaccine distribution through EVI has an average of 5.0% lower in infection cases, 9.4% lower result in death cases, and 3.5% lower in death rates than other distribution methods. © 2021, Springer Nature Switzerland AG.

Explainable Machine Learning for Early Assessment of COVID-19 Risk Prediction in Emergency Departments.

Between January and October of 2020, the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) virus has infected more than 34 million persons in a worldwide pandemic leading to over one million deaths worldwide (data from the Johns Hopkins University). Since the virus begun to spread, emergency departments were busy with COVID-19 patients for whom a quick decision regarding in- or outpatient care was required. The virus can cause characteristic abnormalities in chest radiographs (CXR), but, due to the low sensitivity of CXR, additional variables and criteria are needed to accurately predict risk. Here, we describe a computerized system primarily aimed at extracting the most relevant radiological, clinical, and laboratory variables for improving patient risk prediction, and secondarily at presenting an explainable machine learning system, which may provide simple decision criteria to be used by clinicians as a support for assessing patient risk. To achieve robust and reliable variable selection, Boruta and Random Forest (RF) are combined in a 10-fold cross-validation scheme to produce a variable importance estimate not biased by the presence of surrogates. The most important variables are then selected to train a RF classifier, whose rules may be extracted, simplified, and pruned to finally build an associative tree, particularly appealing for its simplicity. Results show that the radiological score automatically computed through a neural network is highly correlated with the score computed by radiologists, and that laboratory variables, together with the number of comorbidities, aid risk prediction. The prediction performance of our approach was compared to that that of generalized linear models and shown to be effective and robust. The proposed machine learning-based computational system can be easily deployed and used in emergency departments for rapid and accurate risk prediction in COVID-19 patients.